In driver assistance systems with a stereo camera the stereo images are used for calculating a depth image. A depth image is very helpful for many functions of a driver assistance system, including the collision avoidance, the following of other vehicles, etc.
For determining a depth map from stereo images different correlation methods can be used. These algorithms differ in quality and density of the calculated depth map. In addition, the required computing power and the required main memory vary for the calculations.
Basically, the following classes of correlation methods exist:
1. Local correlation methods
2. Global or semi-global methods with specified disparities/labels (discrete optimization methods)
3. Global methods with continuous disparities (continuous optimization methods, e.g. convex optimization).
The advantages and disadvantages of the various groups and methods are not further discussed here.
A disparity means the distance or shift of identical image objects between left and right stereo image.
In a calibrated stereo camera (what can be assumed in the following) only the horizontal distances are to be considered, i.e. the distances in a line.
For practical applications the algorithms from the second group have proved to be particularly suitable. Especially SGM (Semi Global Matching) is regarded as the most functional algorithm for use in real-time systems.
It provides both a high quality of the depth map as well as compared to most other algorithms a low demand of computing power and main memory. On an FPGA (Field Programmable Gate Array) of the latest available driver assistance camera it runs in real time with approximately 16 FPS (frames per second, i.e. images/second). A calculation in real time on a signal processor is not feasible in the foreseeable future.
In fact, for the use of SGM there is currently no alternative which would not involve significant disadvantages. SGM is state of the art and is widely in use.
In the algorithms of the second category and in particular SGM the disparities are determined as integer shifts of the pixels in the image. For this in a first step a comparison operator is used per pixel and disparity. In practice and according to the state of the art, the census operator has proved to be a particularly robust comparison operator.
For this discussion, it is assumed that the right image is the reference frame and (x,y) is an image coordinate. For each pixel P_r (x,y) in the right image the census operator is determined. In the left image the census operator for the pixel P_l(x+d,y) with d=0, . . . , d_max is determined and compared with the census from the right image. This therefore results in a cost measure C (x,y,d) per pixel and disparity. For the entire image, this results in a three dimensional space, which is called cost volume. Based on this cost volume SGM performs an optimization, which determines a disparity per pixel as a result. In doing so, SGM determines by means of an interpolation of the internal costs, which are present for integer and uniformly distributed disparities, a sub-pixel precise disparity in addition to the integer disparity values.
The disparity does not directly indicate the distance z of the next object to the camera (z=0). The connection is reciprocal:z=C1*1/(d+C2)  (1)wherein C1 and C2 are constants. In a calibrated stereo camera C2=0 applies.
C1=f*b depends in a calibrated camera on the following parameters:                f focal length in pixels        b base width        
The accuracy of depth measurement is, therefore, dependent on the depth. In close range a higher accuracy is achieved than in far range. Given a maximum disparity d_max also the minimum determinable distance z_min depends on C1.
From the requirements of a camera system z_min is predefined. A minimal determinable distance must be able to be achieved.
In practice, the accuracy in far range has turned out to be as particularly critical. The accuracy in close range is more than adequate for use in driver assistance systems.
According to the state of art there are several techniques to increase the accuracy. They are shown with their advantages and disadvantages in the following:
1. Interpolation of the Costs
For each pixel the disparity defined by SGM is selected. This disparity is refined while considering the cost of the adjacent disparities. This can be done by a quadratic interpolation with minima-search of the three disparities. Other interpolation schemes (equi-angular fit) are also possible. Details are described in Heiko Hirschmüller, Accurate and Efficient Stereo Processing by Semi-Global Matching and Mutual Information, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 20-26 Jun. 2005, San Diego, Calif., United States, Volume 2, pp. 807-814, improvements can be found e.g. in Stefan K. Gehrig, Uwe Franke, Improving Stereo Sub-Pixel Accuracy for Long-Range Stereo ICCV of 2007.
The advantage of this method is the simple and resource-efficient implementation. However, the disadvantage is that this method often cannot significantly improve the results. One of the main reasons is the effect of the “pixel-locking”, an artifact formation in the sub-pixel interpolation of objects, which are represented by a relatively small number of pixels in the image. Due to the pixel-locking certain interpolated positions (such as e.g. centers or edge points of the pixels) are over-represented.
2. Finer Sampling of the Disparities
In Stefan K. Gehrig, Uwe Franke, Improving Stereo Sub-Pixel Accuracy for Long-Range Stereo ICCV 2007 it is also described that by a finer sampling of the disparities the accuracy of the depth map can be significantly improved.
The cost volume is here resolved finer in the dimension d of the disparity, intermediate steps with 0.5 or 0.25 disparities are inserted. The costs of the intermediate steps are interpolated in the example of the adjacent costs. As a result the cost volume thus contains 2 or 4 times more disparities.
The disadvantage of the finer sampling is that the need for resources, i.e. computing power, storage and memory bandwidth, increases linearly with the number of the disparities.
3. Sub-Pixel Refinement
Starting from an original disparity map the disparities can be refined locally. For this, local correlation methods are used on the two images.
These methods, however, work only in image ranges with a high contrast, i.e. at edges, etc. In practice, it is, therefore, unrealistic to densely refine a disparity map with such methods.
4. Hierarchical Refinement Scheme
In Stefan K. Gehrig, Clemens Rabe, Real-time Semi-Global Matching on the CPU, CVPR 2010 a method is described, in which the disparities in close range are determined with a lower resolution than in far range.
However, this does not apply only for the disparities, but also for the xy-resolution of the pixels. Smaller objects in close range can possibly not be recognized in this way.
In DE 103 10 849 A1 a method for photogrammetric distance and/or position determination is shown, which implements a hierarchical measurement range adjustment. Here, from an original reference and search-gray-scale image pair p new pairs with an increasingly reduced resolution are produced.
In all resolution steps now similarity measures for reference image blocks with equal size search image blocks are determined, the search image blocks being shifted in the respective search gray-scale image pair each in the line direction with a step size of one pixel. The disparity for a reference block is determined by searching sequences of similarity measures for this reference block with regard to extreme values, wherein for all resolution steps except the original resolution step an area each at the beginning of the sequence of similarity measures, which was already detected in the preceding resolution step, is excluded from the search. From the location of the identified extreme value the position of the corresponding object point is determined in a conventional manner.
The disadvantage with this local method is the high expenditure when generating the p image pairs with reduced resolution and the high iteration need for disparity determination.
5. Calculation of Overview and Magnifier Map
In DE 10 2008 015 535 A1 it is described that an overview map and a magnifier card can be calculated separately. The overview map works here at the half resolution over the entire image range and the magnifier card in the full resolution, however, only in a variable section of the image.
The disadvantages of the method are that the magnifier map is not present for the entire image and that the expenditure of resources is doubled by calculating the magnifier map or that in two separate steps first the entire image is calculated with a reduced resolution and then the magnifier map with an increased resolution.